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ABSTRACT 


One  of  the  reasons  military  selection  systems  are  put  into  place  is  to  reduce  high  attrition  rates  in  training. 
Typically  though,  after  the  selection  system  has  experienced  some  operational  use,  the  rejection  and  attrition  rates 
are  observed  to  fluctuate  over  time.  The  question  then  naturally  arises,  “Do  these  fluctuations  represent  some 
substantial  change  in  the  real  world  that  should  be  investigated,  or  do  they  merely  reflect  the  statistical  vagaries 
seen  in  any  small  sample  size?” 

In  this  technical  memorandum,  the  question  just  posed  is  addressed  quantitatively  by  standard  Bayesian 
statistical  techniques.  A  predictive  inference  can  be  made  about  future  frequency  counts  based  on  the  empirical 
data  of  the  past  frequency  counts.  The  uncertainty  surrounding  the  theoretical  rates  mentioned  above  is  handled  m 
the  Bayesian  approach  by  an  averaging  procedure.  The  likelihood  for  any  new  frequencies  conditioned  on  a  given 
rate  parameter  is  averaged  over  the  probability  for  an  acceptable  range  of  these  rate  parameters.  This  probability 
for  the  rate  parameters  is  in  turn  conditioned  on  the  past  frequency  data  and  theoretical  knowledge  about  the  rate 
parameters. 

This  kind  of  analysis  is  helpful  whenever  there  is  concern  that  something  fundamental  might  have  caused  the 
rejection  and/or  attrition  rates  of  the  selection  system  to  change.  For  example,  the  rejection  rates  of  a  given 
selection  system  are  suspected  of  having  changed  dramatically  over  the  past  few  years.  Before  we  can  attempt  to 
track  down  the  cause  of  this  alleged  rate  increase,  we  must  first  establish  that  the  increase  in  frequency  counts  is 
not  simply  due  to  statistical  fluctuations  inherent  in  small  sample  sizes.  It  is  quite  easy  to  be  misled  into  thiniring 
that  a  trend  based  on  relatively  small  numbers  portends  a  significant  change  in  the  underlying  rate  parameter. 

The  techniques  detailed  in  this  report  will  help  researchers  disentangle  sample  size  fluctuations  from  external 
perturbations  to  the  rate  parameter.  In  the  case  of  any  selection  system,  these  techniques  can  be  employed  to 
determine  whether  there  is  a  justification  for  investigating  such  fundamental  changes  as  a  shift  in  the  hardware 
configuration,  a  change  in  the  ability  levels  of  the  candidate  population,  changes  to  the  training  regime,  or  ohangog 
in  the  validity  of  the  current  prediction  algorithm 
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INTRODUCTION 


Militaiy  training  is  a  very  expensive  proposition.  Therefore,  a  concern  for  keeping  costs  down  motivates 
attempts  to  reduce  the  failures  in  training  due  to  poor  performance.  In  militaiy  jargon,  these  training  failures  are 
called  attritions.  One  way  to  reduce  the  attrition  rate  and  lower  costs  is  through  selection  testing  The  hope  is  that 
a  relatively  inexpensive  test  batteiy  can  identify  relevant  traits  that  underly  performance  in  training  and  thus  weed 
out  some  of  the  personnel  who  would  ultimately  fail  in  training. 

The  trade-off  in  using  selection  tests  to  reduce  the  attrition  rate  is  that  some  candidates  are  rejected  by  the  test 
who  otherwise  would  have  succeeded  in  training.  The  question  of  whether  it  is  economically  worthwhile  to 
implement  such  a  selection  test  can  be  answered  by  statistical  decision  theory.  For  a  cost-benefit  example 
concerned  with  the  training  of  Navy  pilots,  see  Blower  f  1], 

In  selection  research,  we  typically  estimate  rejection  rates  and  attrition  rates  for  some  targeted  population  for 
which  the  test  batteiy  was  designed.  These  rates  are  derived  from  some  statistical  model.  The  particular  statistical 
model  examined  here  happens  to  be  discriminant  analysis.  A  primary  output  from  discriminant  analysis  is  a  set  of 
two  normal  curves  for  some  linear  combination  of  test  scores.  Henceforth,  such  a  linear  combination  of  scores 
from  the  test  batteiy  will  be  called  a  composite  score.  One  curve  is  constructed  for  those  who  eventually  fail 
training  with  a  second  curve  for  those  who  eventually  pass  training.  Once  a  threshold  score  is  established,  areas  of 
these  normal  curves  can  be  used  to  estimate  the  rejection  rate  and  the  attrition  rate.  The  detail*  0f  how  these 
numbers  are  derived  are  presented  in  the  next  sectioa 

It  must  be  emphasized  that  these  rates  are  inherently  unobservable  parameters.  Once  a  particular  statistical 
model  has  been  chosen,  it  is  not  a  particular  problem  to  calculate  the  one  number  that  is  the  estimate  of  the 
rejection  rate  or  the  attrition  rate.  We  shall  show  exactly  how  this  can  be  accomplished  in  the  next  sectioa  But 
rejection  rates  cannot  be  measured;  only  the  actual  frequency  of  rejections  can  be  collected  as  data. 

There  is  naturally  some  uncertainly  attached  to  such  rate  parameters  because  of  the  more  fundamental 
uncertainties  that  underly  their  derivatioa  Uncertainties  might  conceivably  arise  due  to  any  of  the  following 
conditions  that  are  assumed  to  impact  the  mean  composite  scores  of  the  two  groups.  The  first  is  statistical  in 
nature.  The  mean  composite  score  for  the  two  groups  is  based  on  a  small  sample  size.  It  would  not  be  surprising 
to  find  that  the  estimate  of  the  mean  composite  score  for  either  or  both  of  the  two  groups  ohangpd  with  a  larger 
sample.  The  other  conditions  are  changes  external  to  the  selection  system.  The  means  of  the  composite  scores 
could  be  drifting  upwards  or  downward s  over  time  due  to  such  causes.  For  example,  the  hardware  components  of 
the  test  battery  may  deteriorate  with  lack  of  maintenance.  New  A/D  cards,  monitors,  and  software  operating 
systems  introduced  to  upgrade  the  system  may  have  an  impact  on  the  composite  scores.  The  drill  level  of  the 
applicants  themselves  could  be  shifting  over  time  as  the  population  from  which  the  applicants  are  drawn  changes. 
Uncertainly  may  exist  concerning  changes  in  the  training  curriculum  that  may  make  it  easier  or  harder  for  an 
applicant  with  a  given  skill  level  to  pass  training. 

All  of  these  influences,  and  others  not  yet  thought  of,  could  conceivably,  acting  separately  or  in  concert,  affect 
the  theoretical  rejection  and  attrition  rates.  It  is  our  lack  of  information  about  these  sources  that  motivates  us  to 
attach  some  measure  of  uncertainty  to  the  one  number  we  calculated  above.  The  only  way  to  optimally  process 
information  in  the  presence  of  uncertainty  is  through  probability  theory,  and  the  best  embodiment  of  probability 
theory  is  the  Bayesian  approach. 

In  stark  contrast  to  these  myriad  uncertainties  listed  above,  there  is  no  uncertainty  in  the  actual  frequency 
counts  (assuming  we  have  been  attentive  enough  to  carefully  record  the  data).  These  frequency  counts  consist  of 
the  actual  number  of  those  rejected  by  the  test  batteiy  and  the  number  Ming  in  training.  The  question  then 
becomes  whether  the  currently  believed  theoretical  rates  are  in  consonance  with  the  empirical  frequency  counts. 

In  this  technical  memorandum,  the  question  just  posed  is  addressed  quantitatively  by  standard  Bayesian 
statistical  techniques.  A  predictive  inference  can  be  made  about  future  frequency  counts  based  on  the  pmpirir^i 
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data  of  the  past  frequency  counts.  The  uncertainty  surrounding  the  theoretical  rates  mentioned  above  are  handled 
in  the  Bayesian  approach  by  an  averaging  procedure.  The  likelihood  for  any  new  frequencies  conditioned  on  a 
given  rate  parameter  is  averaged  over  the  probability  for  an  acceptable  range  of  these  rate  parameters.  This 
probability  for  the  rate  parameters  is  in  turn  conditioned  on  the  past  frequency  data  and  theoretical  knowledge 
about  the  rate  parameters.  These  statements  are  made  precise  in  upcoming  sections  of  the  report. 

Those  who  are  in  charge  of  training  programs  are  understandably  concerned  about  managing  the  flow  of 
personnel.  Fluctuations  in  the  number  of  rejections  and  attrition  from  a  selection  system  often  raise  a  red  flag  of 
concern  The  question  then  naturally  arises,  “Do  these  fluctuations  represent  some  substantial  change  in  the  real 
world  that  should  be  investigated,  or  do  they  merely  reflect  the  statistical  vagaries  seen  in  any  small  sample  size?” 

The  kind  of  analysis  outlined  here  is  helpful  whenever  there  is  concern  that  something  fundamental  might  have 
caused  the  rejection  rates  and/or  attrition  rates  of  the  selection  system  to  change.  Before  we  can  attempt  to  track 
down  the  cause  of  this  alleged  rate  increase,  we  must  first  establish  that  the  increase  or  decrease  in  frequency 
counts  cannot  be  explained  by  statistical  fluctuations  inherent  in  small  sample  sizes.  It  is  quite  easy  to  be  misled 
into  thinking  that  a  “trend”  based  on  relatively  small  numbers  portends  a  significant  change  in  the  underlying  rate 
parameter.  The  quantitative  ideas  developed  here  are  applied  to  an  actual  military  selection  system;  the  Landing 
Craft  Air  Cushion  Vehicle  (LCAC)  Selection  System  for  Operators  and  Engineers.  The  results  of  this  analysis  are 
presented  in  a  companion  paper  [2]. 

More  generally,  one  would  like  to  monitor  the  rejection  rates  and  attrition  rates  from  a  given  selection  system 
to  detect  if  any  changes  have  occurred  that  should  prompt  a  review  of  the  selection  system.  This  is  the  same 
philosophy  behind  industrial  quality  control  procedures  that  seek  to  identify  the  causes  of  glitches  in  a 
manufacturing  process. 

Initial  information  about  the  rate  parameters 

This  section  provides  an  illustration  of  how  information  about  the  theoretical  rate  parameters  can  be  extracted 
from  a  given  statistical  model.  The  two  rate  parameters  we  are  concerned  with  here  are  the  rejection  rate  and  the 
attrition  rate  that  follow  from  operational  use  of  the  selection  system.  Please  refer  to  Fig.  1  during  the  ensuing 
discussion 

At  the  top  of  the  figure  two  normal  curves  are  drawn  The  curve  to  the  left  represents  the  distribution  of  test 
battery  scores  for  those  candidates  who  fail  training,  and  the  second  curve  to  its  right  represents  the  distribution  of 
test  battery  scores  for  those  candidates  who  pass  training.  The  x-axis  is  actually  a  composite  score  based  on  a 
linear  weighting  of  some  number  of  individual  tests  from  the  overall  test  battery.  The  y-axis  shows  the  usual 
probability  density  function  values  for  a  normal  distribution  with  the  stated  mean  and  standard  deviation. 

The  means  and  standard  deviations  (a)  for  the  composite  score  are  shown  for  the  two  groups  called  FAIL  and 
PASS.  These  means,  because  they  are  computed  from  a  small  sample  size,  may  differ  from  the  true  value  of  the 
mean  and  this  is  indicated  by  placing  a  “hat”  above  fi.  The  discriminant  analysis  procedure  tries  to  construct  a 
composite  score  such  that  the  means  of  the  two  groups  are  as  far  apart  as  possible.  As  part  of  this  procedure,  the  a 
of  the  composite  score  is  made  equal  to  1.00  for  both  groups.  The  threshold  score  is  established  through  statistical 
decision  theory  and,  in  this  example,  is  placed  at  a  value  of  +0.20.  A  candidate  who  scores  below  this  threshold, 
or  cut-off  score,  is  predicted  to  be  a  failure  in  training,  while  any  candidate  who  scores  above  the  threshold  score 
is  predicted  to  be  a  success  in  training.  For  further  details  of  this  model  see  Blower  [3]. 

Given  the  two  normal  curves  and  the  placement  of  the  cut-off  score,  four  areas  can  be  defined.  These  four 
areas  represent  the  four  possible  combinations  of  the  two  true  states  of  the  world,  1)  candidate  actually  passes 
training  and  2)  candidate  actually  fails  training,  with  the  two  decisions  that  are  made,  1)  predict  candidate  passes 
and  2)  predict  candidate  fails.  There  are  thus  two  ways  to  be  right  and  two  ways  to  be  wrong. 

From  the  normal  curves  and  the  placement  of  the  threshold  score,  we  can  assign  percentages  to  each  one  of 
these  four  areas.  Since  the  estimated  mean  of  the  FAIL  distribution,  p  =  —0.10,  is  .30a  from  the  threshold  score, 
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Figure  1:  'Die  initial  rejection  rate  and  attrition  rate  estimates  for  the  selection  system  as  derived  from  a  statistical 
model  and  information  from  the  R&D  phase. 
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the  area  in  the  larger  portion  of  the  FAIL  distribution  is  61.79%.  The  area  in  the  smaller  portion  of  the  FAIL 
distribution  must  therefore  be  38.21%.  Likewise,  since  the  mean  of  the  PASS  distribution,  ji  =  +0.90,  is  .70cr 
from  the  threshold  score,  the  area  in  the  larger  portion  of  the  PASS  distribution  is  75.80%.  The  area  in  the  smaller 
portion  of  the  PASS  distribution  is  24.20%.  These  numbers  can  be  found  in  any  table  of  the  normal  distribution. 

The  2x2  matrix  found  in  the  center  of  the  figure  contains  the  actual  frequencies  for  the  four  areas  of  the 
normal  curve  as  obtained  from  the  research  and  development  (R&D)  phase  prior  to  the  operational  implementation 
of  the  selection  system.  It  shows  the  number  of  candidates  who  fell  into  the  two  categories  of  a  correct  prediction, 
that  is,  45  and  19,  and  the  number  of  candidates  who  fell  into  the  two  categories  for  a  wrong  prediction,  that  is,  15 
and  11.  Of  the  total  of  90  subjects  who  participated  in  the  R&D  experiments,  60  subjects  eventually  passed 
training  and  30  subjects  eventually  failed  training. 

Given  this  context,  we  are  now  ready  to  estimate  the  rejection  rate  and  the  attrition  rate.  The  estimated 
rejection  rate  is  the  number  of  candidates  who  score  below  the  threshold  and  are  therefore  predicted  to  fail,  divided 
by  the  total  number  of  candidates  tested.  The  attrition  rate  is  the  number  of  candidates  that  actually  fail  training 
from  the  number  that  went  into  training  because  their  scores  were  above  the  cut-off  score  and  were  therefore 
predicted  passes.  This  calculation  is  shown  at  the  bottom  of  the  figure.  The  rejection  rate  is  estimated  at  37.78% 
and  the  attrition  rate  is  estimated  at  19.64%. 

The  composite  mean  scores  of  +0.90  and  —0.10  shown  in  Fig.  1  for  the  PASS  and  FAIL  groups  were 
constructed  from  a  relatively  small  sample  during  the  R&D  phase  of  the  selection  system  development.  From 
sample  size  considerations  alone,  we  are  uncertain  about  these  mean  values  and  would  expect  them  to  fluctuate. 

For  the  same  reason,  the  counts  in  the  2  x  2  matrix  are  subject  to  fluctuation  as  well.  So,  all  in  all,  our  estimates 
of  the  rejection  rate  and  the  attrition  rate  are  fraught  with  some  level  of  uncertainty.  Probability  theory,  and  more 
specifically  the  Bayesian  formalism,  are  employed  to  deal  with  this  uncertainty  in  an  optimal  fashion. 

The  uncertainty  in  rejection  rates  and  attrition  rates  due  to  small  sample  sizes 

As  shown  in  Fig.  1  above,  we  formed  an  estimate  of  the  rejection  rate  and  the  attrition  rate  based  on  the 
relatively  small  sample  that  was  available  to  us  during  R&D.  The  rejection  rate  was  a  little  less  than  38%,  and  the 
attrition  rate  was  just  below  20%.  Most  likely,  if  we  were  to  take  another  sample,  the  estimates  of  /xpa33  and 
UFaii  would  change.  In  addition,  the  number  of  candidates  actually  passing  and  failing  in  training  would  fluctuate 
as  well  with  another  sample. 

As  a  numerical  example,  consider  Figs.  2  and  3  which  are  just  like  Fig.  1  except  that  we  consider  plausible 
changes  to  the  estimates  of  fj,pass  and  fiFa u  by  the  discriminant  analysis  program  on  different  samples.  In  Fig.  2 
we  show  a  new  sample  that  gives  pLFail  =  -0.20  and  fiPasa  =  +0.80.  Notice  that  the  discriminability  of  the  test 
battery  remains  the  same  because  the  separation  between  the  PASS  and  FAIL  means  is  still  1.00.  Also,  the  number 
of  candidates  actually  passing  training  is  lowered  to  55  and  the  number  failing  raised  to  35  to  exhibit  sampling 
fluctuations  here  as  well.  Now  our  estimate  of  the  rejection  rate  is  raised  to  above  42%  and  the  attrition  rate  to 
above  23%  when  compared  to  the  previous  estimates. 

In  Fig.  3,  the  change  in  the  estimate  of  npaas  and  nFau  goes  in  the  opposite  direction  with  yet  another 
hypothesized  sample  of  90  candidates.  p,Fau  now  equals  0.00  and  p.pas,  equals  +1.00.  The  sampling  variability 
for  the  actual  number  of  passes  and  fails  also  changes  in  the  opposite  direction  with  65  passes  and  25  fails.  In  this 
situation,  the  estimate  of  the  rejection  rate  is  lowered  to  about  31%,  and  the  attrition  rate  is  lowered  to  a  little  less 
than  18%.  The  reader  can  imagine  many  other  possible  combinations  arising  from  repeated  sampling  of  90 
subjects  and  the  resulting  distribution  of  estimated  rejection  rates  and  attrition  rates. 

Reasonable  supposition  about  changes  in  the  values  to  npa»»  and  ixFau  and  the  actual  passes  and  fails  in 
training  from  different  samples  of  the  same  small  size  leads  to  different  estimates  for  the  rejection  rate  and  attrition 
rate.  Therefore,  due  to  the  restrictions  of  small  sample  sizes,  we  are  uncertain  about  the  exact  value  of  the  rejection 
and  attrition  rate. 
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Figure  2:  Plausible  changes  to  the  estimates  of  the  mean  composite  scores  as  well  as  to  the  numbers  actually  passing 
and  failing  training  for  a  new  sample  size.  These  kind  of  changes  result  in  raising  the  rejection  and  attrition  rate 
estimates  as  calculated  in  Fig.  1. 
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Figure  3:  Plausible  changes  to  the  estimates  of  the  mean  composite  scores  as  well  as  to  the  numbers  actually  passing 
and  failing  training  for  yet  a  third  hypothesized  sample  size.  These  kind  of  changes  result  in  lowering  the  rejection 
and  attrition  rate  estimates  as  calculated  in  Fig.  1. 
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The  only  way  to  capture  uncertainty  and  deal  with  it  quantitatively  is  through  probability  theory.  The  Bayesian 
formalism  to  be  outlined  in  the  next  few  sections  uses  probability  theory  to  develop  its  core  concepts  of  prior 
probability,  likelihood,  posterior  probability,  and  predictive  inferences.  In  the  next  section,  we  begin  by  rfismsging 
the  prior  probability  distribution.  It  is  within  this  distribution  that  we  are  able  to  ejqrress  the  uncertainty  about  the 
rejection  and  attrition  rate  parameters  just  outlined. 

Inserting  information  into  the  prior  distribution 


Let  6  stand  for  a  rate  parameter  such  as  the  rejection  or  attrition  rate.  The  prior  distribution  for  0,  p(9), 
captures  our  state  of  knowledge  about  a  rate  parameter  before  arty  data  are  gathered,  but,  nonetheless,  imict  be 
conditioned  on  whatever  other  information  may  be  at  our  disposal.  In  this  case,  this  other  information  is  the 
expected  rate  of  rejection  and  expected  attrition  rate  stemming  from  the  discriminant  analysis  model  and  the  prior 
data  gathered  during  the  R&D  phase  of  the  selection  tystem.  The  other  information  we  wish  to  insert  into  the  prior 
distribution  is  the  base-level  uncertainty  about  these  two  expected  rates.  The  prior  distribution  thus  represents  our 
tentative  working  hypothesis  based  on  all  the  information  in  our  possession  until  actual  frequency  data  from  the 
operational  selection  system  force  us  to  modify  it  through  Bayes’s  Formula. 

From  a  technical  mathematical  standpoint,  the  beta  distribution  is  a  convenient  prior  distribution  to  capture 
information  about  parameters  like  the  rate  parameter  6  that  can  vary  between  0  and  1.  To  fix  the  precise  shapp  and 
location  of  the  beta  distribution,  we  must  specify  two  hyperparameters.  These  two  hyperparameters  have 
traditionally  been  given  the  labels  of  a  and  0,  both  of  which  must  be  greater  than  0. 

There  are  formulas  that  tell  us  the  mean  and  standard  deviation  of  any  beta  distribution  as  a  function  of  a  and 
0.  The  mean  is  especially  simple, 


m  = 


a 

a  +  0 


The  standard  deviation  is  slightly  more  complicated,  but  still  easily  computed  as, 


(1) 


SDmmh*+»‘Z+fjij  m 

We  want  to  choose  values  for  a  and  0  that  reflect  our  state  of  knowledge  about  the  rate  parameter  6  based  on  the 
description  given  in  the  second  section.  For  example,  we  think  that  the  mean  of  the  rejection  rate  parameter  should 
be  about  38%.  Likewise,  the  mean  of  the  attrition  rate  parameter  is  about  20%.  These  estimates  are  based  on  the 
one  rather  small  sample  of  candidates  that  we  were  able  to  test  during  R&D.  But  we  are  also  cognizant  of  some 
uncertainty  attached  to  these  estimates.  Assume  that  we  are  willing  to  let  the  standard  deviation  of  6  be  a  value 
such  that  E(6)  ±  2 SD{6)  captures  most  of  our  uncertainty  about  0. 

Let  us  first  assign  a  prior  probability  for  the  rejection  rate  parameter  called  6r.  If  we  let  a  =  19  and  0  —  31, 
then  using  Equations  (1)  and  (2)  we  find  that. 


m) 


19 

19  +  31 


=  .38 


SD(6r ) 


Therefore,  if  E(0r)  =  .38  and  SD(6r)  =  .0680, 

Rejection  rate 


19x31 
502  x  51 


.0680 


E{9r)  ±2SD{6r) 
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Figure  4:  A  sketch  of  the  beta  distribution  that  serves  as  the  repository  for  a  state  of  knowledge  about  the  true 
rejection  rate  parameter  based  solely  on  prior  R&D  data  and  a  discriminant  analysis  model. 


=  .38  ±  2(.0680) 


=  .2440  to  .5160 


and  our  state  of  knowledge  is  that  the  hue  rejection  rate  probably  lies  in  the  range  from  about  25%  to  almost  52%. 
This  is  the  price  we  have  to  pay  for  uncertainty  about  the  unobservable  rejection  rate  parameter.  Using  larger 
values  of  a  and  /3  than  those  given  above  will  reduce  the  uncertainty  about  E(0),  and  conversely,  using  smaller 
values  will  increase  the  uncertainty.  Fig.  4  contains  a  sketch  of  the  beta  distribution,  which  serves  as  the  prior 
probability  for  the  true  rejection  rate  parameter.  This  curve  encapsulates  our  uncertainty  about  6  before  any 
frequency  data  has  been  collected. 

Now,  using  the  very  same  techniques,  we  can  assign  a  prior  probability  to  the  attrition  rate  parameter,  0a.  In 
this  case,  if  we  let  a  =  20  and  =  80,  then  using  Equations  (1)  and  (2)  we  find  that, 

E^6a)  =  20  +  80 
=  .20 


SD(Oa )  = 


/ 


20x80 
1002  x  101 


.0398 


Therefore,  if  E(6a)  =  .20  and  SD(9a)  =  .0398, 

Attrition  rate  =  E(0a)±2SD(6a) 


=  .20  ±  2(.0398) 


=  .1204  to  .2796 


8 


Probability 


Figure  5:  A  sketch  of  the  beta  distribution  that  serves  as  the  repository  for  a  state  of  knowledge  about  the  true 
attrition  rate  parameter  based  solely  on  prior  R&D  data  and  a  discriminant  analysis  model. 


our  state  of  knowledge  is  that  the  true  attrition  rate  probably  lies  in  the  range  from  about  12%  to  almost  28%. 
Since  it  appeared  from  our  numerical  examples  of  the  last  section  that  the  attrition  rate  parameter  did  not  vary  over 
as  large  a  range  as  the  rejection  rate  parameter,  we  use  larger  values  for  a  and  (3.  This  results  in  a  narrower  width 
for  the  prior  probability  for  6a  and,  as  such,  reflects  reduced  uncertainty  about  the  values  0a  could  reasonably 
assume.  Figure  5  shows  the  prior  probability  density  function  assigned  to  the  attrition  rate  parameter.  Notice  that 
the  mean  is  shifted  towards  rite  left  and  the  curve  possesses  a  narrower  width  because  we  are  less  uncertain  of  the 
range  of  values  that  it  might  take  oa 

How  the  state  of  knowledge  about  the  rate  parameters  change  with  data 

After  the  selection  system  is  transitioned  from  R&D  to  an  operational  system,  frequency  data  are  collected  on 
the  number  of  rejections  and  the  number  of  attritions.  This  additional  information  can  be  used  to  update  the  state 
of  knowledge  about  the  true  rejection  and  attrition  rate  parameters.  This  updating  is  accomplished  through  an 
application  of  Bayes’s  Formula.  The  major  result  from  Bayes’s  Formula  is  a  revision  to  the  prior  distributioa  This 
updated  probability  distribution  is  called  the  posterior  distribution  because  it  exists  after  the  data  have  been 
incorporated  as  extra  informatioa 

The  posterior  distribution,  when  compared  to  the  prior  distribution,  conditions  on  two  more  pieces  of  additional 
information  that  are  assumed  to  be  true.  The  prior  distribution  conditions  on  the  assumed  troth  of  a  and  /?,  while 
the  posterior  distribution,  say  for  the  rejection  rate  parameter,  conditions  not  only  on  a  and  /?,  but  also  on  the 
number  of  candidates  actually  tested  and  the  number  of  candidates  rejected  because  of  poor  performance  on  the 
test  battery. 

Say  that  frequency  data  for  rejections  and  attritions  have  been  collected  from  the  operational  selection  system 
for  the  past  5  years.  For  brevity,  we  will  discuss  only  rejection  rates,  but  the  same  argument  applies  as  well  to 
attrition  rates.  The  notation  for  these  empirical  frequency  counts  that  have  been  gathered  in  the  past  is  as  follows: 
n  is  the  total  number  of  candidates  that  have  been  tested  in  the  past;  y  is  the  actual  number  of  candidates  that  were 
rejected  because  their  scores  fell  below  the  threshold  score;  and  p(6r  |y,  n,  a,  (3)  is  the  notation  for  the  posterior 
distribution  for  the  rejection  rate  parameter  conditioned  on  the  past  frequency  counts.  The  posterior  probability  is 
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found  by  an  application  of  Bayes’s  Formula  that  relies  on  the  likelihood  for  the  past  data  and  the  prior  probability 
of  0r. 


p{6r\y,n,  a,  0) 


P(y|gr,n)  xp(flr la,/?) 

Jo  d6r  P(y\Or,n)  x  p(6r\a,  (3) 


(3) 


Table  1  presents  a  simulation  of  the  number  of  candidates  tested  and  the  number  rejected  by  the  test  battery  for 
each  of  the  five  years  after  the  operational  implementation  of  the  selection  system.  The  operational  selection 
system  tests  about  100  candidates  per  year.  The  last  two  columns  contain  a  cumulative  record  of  the  number 
tested,  n,  and  the  number  rejected,  y. 


Table  1:  The  simulated  number  of  candidates  rejected  by  the  selection  system  for  the  first  5  years  after  becoming 
operational. 


Year 

Number  tested 

Number  rejected 

n 

y 

1 

100 

32 

100 

32 

2 

95 

30 

195 

62 

3 

106 

46 

301 

108 

4 

102 

35 

403 

143 

5 

92 

29 

495 

172 

If  we  wish  to  incorporate  these  frequency  counts  for  rejections  over  the  past  5  years  into  the  information 
already  contained  in  the  prior  for  0r,  we  can  set  n  =  495  and  y  =  172  and  calculate  the  posterior  distribution.  The 
posterior  distribution  can  be  calculated  and  plotted  just  like  the  prior  distnbution  Again,  the  appropriate  formula 
for  calculating  the  posterior  distribution  is  contained  in  Reference  [4]  as  Equation  (15).  Fig.  6  shows  the  posterior 
distribution  as  conditioned  on  the  empirical  data  from  Table  1  overlaid  onto  the  prior  distribution  taken  from  Fig.  5. 
The  additional  information  in  the  data  has  changed  the  state  of  knowledge  and  therefore  the  uncertainty  about  the 
true  rejection  rate.  The  posterior  curve  is  shifted  to  the  left  with  a  mean  closer  to  E(6)  =  .35,  but  it  is  more 
concentrated  around  a  narrower  range  of  values.  We  are  less  uncertain  about  6  after  having  collected  these  data. 
Before  the  empirical  data  were  gathered,  the  state  of  knowledge  about  0  was  that  it  probably  ranged  from  .23  to  .52 
with  a  mean  of  .38.  After  incorporating  the  data  via  Bayes’s  Formula,  the  state  of  knowledge  about  9r  is  updated. 
The  rejection  rate  parameter  has  been  concentrated  into  a  highly  probable  range  of  .32  to  .38  with  a  mean  of  .35. 

The  Bayesian  formalism  for  predictive  inference 

Of  great  practical  interest  to  the  user  of  the  selection  system  is  the  question,  “How  can  one  predict  the  number 
of  candidates  that  will  be  rejected  by  the  test  batteiy  in  some  future  time  period?”  To  answer  this  question,  the 
Bayesian  formalism  includes  a  fourth  concept,  predictive  inference. 

The  ingredients  for  the  predictive  inference  consist  of  the  likelihood  for  arty  future  number  of  rejections  and  the 
posterior  based  on  empirical  data  consisting  of  past  frequency  counts.  The  posterior  probability,  by  definition,  also 
contains  the  information  discussed  in  the  second  section  and  which  was  originally  placed  into  the  prior.  Before  we 
present  the  Bayesian-inspired  formula  for  producing  a  predictive  inference,  we  will  spend  some  time  reviewing  the 
notation.  The  predictive  inference  formula  will  then  be  used  to  generate  graphs  that  speak  to  the  question  posed  at 
the  beginning  of  this  section 

In  the  situation  we  are  examining  here,  we  are  concerned  with  unobserved,  or  theoretical,  rate  parameters,  viz., 
the  true  rejection  rate  and  the  true  attrition  rate.  We  let  9r  and  0a  stand  for  such  rate  parameters.  As  a  rate 
parameter,  0  takes  on  values  between  0  and  1.  We  have  already  seen  that  the  prior,  p{0),  encapsulates  the 
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Rejection  Rate  Parameter 

Figure  6:  A  comparison  of  the  posterior  distribution  for  6r  with  the  prior.  The  shift  in  location  and  the  narrowing 
of  the  curve  represents  the  intact  of  the  empirical  data 


uncertainty  surrounding  the  true  rate  parameter  before  we  collect  any  actual  frequencies  from  the  operational 
system.  Using  a  and  /?  allowed  us  to  specify  our  prior  state  of  knowledge  about  the  rate  parameter.  We  leveraged 
a  and  (3  to  translate  what  we  knew  about  0  from  the  particular  statistical  model  we  happened  to  be  using  in 
addition,  in  what  is  quite  an  important  feature  of  this  methodology,  we  included  the  uncertainty  surrounding  0 
arising  from  the  fluctuations  due  to  small  sample  size.  This  uncertainty  about  0  was  reduced  when  we  started 
collecting  data  from  the  operational  system.  The  number  of  rejections,  y,  from  the  total  number  of  candidates 
tested,  n,  provided  us  with  additional  information  that  we  processed  into  the  posterior  distribution. 

Finally,  we  let  z  stand  for  the  number  of  candidates  that  will  be  rejected  by  the  test  battery  in  some  future  time. 
period  of  interest,  say  the  upcoming  year.  N  represents  the  future  total  number  of  candidates  that  would  be  tested 
in  the  upcoming  year.  Later  in  this  report,  we  examine  values  of  N  equal  to  10,  50  and  100.  In  projecting  the 
future  number  of  rejections,  N  can  assume  any  sample  size  that  might  be  of  interest.  Values  for  2  can  potentially 
range  from  0, 1, 2,  all  the  way  up  to  N. 

L(z\0,  N)  is  the  likelihood  of  obtaining  z  =  0, 1, 2,  •  •  • ,  N  rejections  in  some  future  sample.  The  likelihood  is 
conditioned  on  some  value  of  6,  which  is  taken  to  be  hue,  and  the  given  value  of  N.  For  example, 

L(z  =  17\0r  =  .30,  N  =  50)  is  the  likelihood  of  obtaining  17  rejections  from  a  sample  of  size  50  when  the  true 
underlying  rate  parameter,  in  this  case  the  rejection  rate,  is  equal  to  .30. 

Two  fundamental  axioms  of  probability  theory  are  used  to  link  the  elements  just  discussed,  namely,  the 
likelihood  of  a  future  number  of  rejections  and  the  posterior  probability  of  the  rate  parameter,  into  the  formula  for 
predictive  inference. 


P(z\N,y,n,a, 


d0r  L{z\0r,  N)  p(0r\y,  n, a,/3) 


(4) 


For  example,  P(z  =  17| N  =  50,  y  =  172,  n  =  495,  a  =  19,  /?  =  31)  will  tell  us  the  probability  of  Staining  17 
rejections  in  a  future  sample  of  size  50  given  that  data  exist  where  172  known  rejections  occured  in  a  known 
sample  size  of  495,  and,  in  addition,  where  we  have  specified  prior  knowledge  about  the  theoretical  rate  parameter 
in  a  and  (3.  Working  out  the  mathematical  consequences  of  Equation  (4)  is  a  well-known  exercise  in  Bayesian 
statistics.  All  the'  tedious  details  are  spelled  out  in  a  technical  report  (Blower  [4]),  which  also  derives  a 
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computational  formula  that  can  be  inserted  into  a  computer  program 

The  important  point  is  that,  given  values  for  N,  z,  y,  n,  a  and  ft,  such  a  program  can  compute  the  predictive 
probability.  We  shall  use  a  more  concise  notation  for  the  predictive  inference,  P{z),  that  deletes  the  explicit 
reference  to  the  fact  that  N,  y  and  n  are  known  and  that  a  and  ft  are  assumed  to  take  on  given  values.  Repeating, 
P{z)  is  the  Bayesian  state  of  knowledge  for  obtaining  2  rejections  from  a  future  sample  of  size  W  based  on  past 
data  where  y  rejections  actually  occurred  in  a  sample  of  size  n,  and  where  a  and  ft  capture  any  prior  knowledge 
about  9,  the  rate  parameter. 

Numerical  example  of  how  to  calculate  2 

Before  showing  the  graphs  of  the  predictive  probability  for  the  number  of  rejections  for  various  future  sample 
sizes  N,  the  computational  formula  will  be  illustrated  for  one  set  of  values  of  N,  z,  y,  n,  a  and  ft.  First,  N  is  a 
value  that  we  are  free  to  choose  because  it  happens  to  be  of  intrinsic  interest  in  the  particular  problem  that 
confronts  us.  Suppose  that  100  candidates  are  projected  to  be  tested  in  the  upcoming  year,  and  we  are  interested  in 
whether  the  rejection  rate  is  holding  steady  at  what  our  theoretical  model  told  us  it  should  be.  Halfway  through 
testing,  we  would  like  to  assess  the  selection  system  for  any  untoward  circumstances.  We  therefore  let  N  =  50  and 
calculate  P(z)  for  all  values  of  z  from  2  =  0  to  z  =  50,  yielding  a  discrete  probability  distribution  for  51  values. 

Fixed  values,  n  and  y,  come  from  previous  data.  For  the  present  numerical  example,  we  will  use  the  data  given 
in  Table  1  to  fill  in  the  values  for  y  and  n.  Therefore,  n  =  495  and  y  =  172  because,  as  mentioned  above,  there 
were  172  rejections  from  the  495  candidates  tested. 

All  the  values  needed  to  perform  the  calculation  are  now  present.  If  2  =  17  and  N  =  50,  we  want  to  find  the 
probability  of  obtaining  17  rejections  from  the  selection  system  from  a  future  sample  of  size  50.  The  inference  will 
be  based  on  data  indicating  that  in  the  past  5  years  172  candidates  were  rejected  out  of  the  495  that  were  tested.  In 
addition,  we  inserted  initial  information  about  the  rejection  rate  through  a  and  ft,  which  indicated  that  we  thought 
that  the  mean  rejection  rate  was  about  .38  with  an  attached  95%  confidence  interval  stretching  from  .25  to  .52. 
This  range  of  values  reflected  our  initial  uncertainty  about  the  value  .38  for  the  rejection  rate  parameter  because  we 
took  notice  of  the  small  sample  size  in  the  R&D  phase  and  built  this  into  our  prior  distributioa 

The  formula  to  compute  the  predictive  inference  contained  in  Equation  (4)  is  taken  from  Blower  [4], 

P(z)  =  f(N,z,n,y,a,ft )  (5) 

(6) 


where 

x  =  In  TV'S -ln(N- 2)!  -  lnz!  +  ln(a  +  /?  +  n-  1)!  —  ln(y  +  a—  1)! 

—  ln(n  -  y  +  ft  -  1)!  +  ln(z  +  y  +  a  —  1)! 

+ln(N  —  z  +  n  —  y  +  ft—  1)!  —  ln(a  +  /?+  N  +  n—  1)!  (7) 

Table  2  lists  the  necessary  computations  of  the  log  factorials  needed  to  solve  Equations  (6)  and  (7).  Most  computer 
software  designed  for  numerical  calculations  will  possess  built-in  functions  for  determining  the  natural  log 
transform  of  factorials.  Sometimes,  the  log  transform  of  the  gamma  function  is  provided  instead. 

Substituting  these  numbers  into  Equations  (6)  and  (7)  yields, 

P(z)  =  /(IV  =  50,  n  =  495, 2/ =  172,  z  =  17,  a  =19,/?  =  31) 
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Table  2:  A  listing  of  all  the  numerical  values  and  the  log  transforms  of  the  factorial  functions  needed  to  calculate 
a  predictive  probability  for  the  example  given  in  the  test 


Terms  in  Eq.  (7) 

Value  Col.  1 

In  of  terms  in  Eq.  (7) 

Value  Col.  3 

N 

50 

In  N\ 

148.48 

N-z 

33 

ln(N-z)\ 

85.05 

z 

17 

lnz! 

33.51 

a  +  /3  +  n  —  l 

544 

ln(a  +  0  +  n  —  1)! 

2866.70 

y  +  a-  1 

190 

In (y  +  a  -  1)! 

810.48 

n-y+0-1 

353 

ln(n  -  y  +  0  -  1)! 

1721.72 

z+y+a- 1 

207 

ln(z  +  y  +  a  —  1)! 

900.46 

N  — z+n— y  +  0  — 1 

386 

In  (N  -z  +  n  —  y  +  0  —  1)! 

1916.85 

a  +  (3  +  N  +  n  —  1 

594 

ln(a  +  0  +  N  +  n  —  l)l 

3203.92 

x  =  —2.19 
P(z)  =  .1121 

Therefore  the  probability  of  obtaining  17  rejections  from  a  future  sample  size  of  50  is  a  little  over  11%. 

Examples  of  the  predictive  probability  distribution 

While  the  proceeding  section  showed  how  one  value  of  z  is  calculated,  a  computer  program  can  calculate  z  for 
all  values  of  z  from  0  to  N.  The  program  can  then  produce  a  bar  graph  giving  the  predictive  probability  at  each 
one  of  these  z  values.  The  uncertainty  about  the  frequency  of  future  rejections  for  any  given  sample  size  is 
captured  by  this  predictive  probability  distribution  The  graph  can  be  consulted  to  obtain  an  idea  of  which 
frequency  counts  are  compatible  with  this  distribution  because  they  fall  into  an  high  probability  area  and  which 
frequency  counts  seem  to  be  extreme  because  they  fall  into  an  extremely  low  probability  area  The  usual  criterion 
will  be  used  that  accepts  a  frequency  count  that  falls  within  the  95%  region  of  the  predictive  inference  distribution 
as  compatible,  while  a  frequency  that  falls  into  the  5%  region  is  viewed  as  extreme. 

This  section  contains  three  graphs  of  the  predictive  probability  distributions  for  sample  sizes  of  N  =  10,  50, 
and  100.  These  values  of  N  allow  one  to  monitor  the  frequency  counts  at  an  early,  mid,  or  final  stage  of  the 
upcoming  year.  These  graphs  would  be  used  to  monitor  the  performance  of  the  selection  system  after  Year  5  by 
assessing  where  the  actual  number  of  rejections  in  a  sample  of  size  N  fits  into  the  predictive  distribution  We 
begin  by  showing  Figure  7  which  contains  the  graph  for  a  sample  of  size  N  —  10.  As  portrayed  in  this  Figure  and 
the  next  two,  the  predictive  probability  distributions  are  discrete  distributions  with  2  =  0, 1,  •  *  *  N  along  the  x-axis. 
The  y-axis  indicates  the  probability  for  any  particular  z  with  the  probability  calculated  by  Equations  (6)  and  (7). 
The  distributions  are  generally  symmetrical  with  the  mode  of  the  distribution,  that  is,  the  most  probable  value  of  2, 
located  in  the  center  of  the  distribution  mass.  Some  slight  positive  skewness  however  is  evident  for  small  sample 
sizes  such  as  N  =  10. 

Looking  at  Fig.  7,  the  most  probable  value  for  the  number  of  rejections  for  a  future  sample  size  of  N  =  10  is 
z  —  3  rejections  with  a  probability  just  about  25%.  If  we  had  to  pick  one  value  as  our  guess  for  the  number  of 
rejections  this  would  be  it.  Of  course,  this  corresponds  nicely  to  an  intuitive  guess  based  on  the  past  empirical  data 
and  the  theoretical  estimate  for  the  rejection  rate  parameter.  All  of  this  was  captured  in  the  posterior  probability 
distribution,  which  peaked  at  6r  =  .35.  The  Bayesian  formalism  never  violates  common  sense  in  cases  where  the 
answer  is  obvious.  Unfortunately,  unaided  intuition  is  not  so  reliable  in  assessing  the  probabilities  of  other  possible 
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Figure  7:  The  predictive  probability  graph  showing  the  probabUity  of  the  number  of  rejections  for  the  selection 
system  for  a  sample  size  of  N  =  10.  This  probability  distribution  is  based  on  past  empirical  data  of  172  rejections 
out  of  495  candidates  tested  and  a  theoretical  judgment  of  the  rejection  rate  based  on  a  beta  distribution  with  a  mean 
of  .38. 
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rejection  numbers. 

It  is  important  to  realize  that  probability  distributions  reflect  the  uncertainty  about  any  one  value,  and  it  appears 
that  there  is  a  lot  of  uncertainty  for  such  a  small  sample  size.  For  example,  although  3  rejections  is  the  most 
probable  number  of  rejections,  nearby  values  cannot  be  ruled  out  1,  2,  4,  5,  and  6  rejections  are  also  not  unlikely. 
The  number  of  rejections  that  we  deem  unlikely  occurs  at  the  ends  of  the  distribution.  Therefore,  we  would 
consider  no  rejections,  occurring  at  the  left  end  of  the  distribution,  and  7  through  10  rejections,  occurring  at  the 
right  end  of  the  distribution,  as  improbable.  The  probability  of  0  rejections  is  .0140,  and  the  probability  of  7,  8,  9 
or  10  rejections  is  .0221,  .0046,  .0006,  and  .00003,  respectively.  Together  these  five  extreme  values  make  up  4.1% 
of  the  total  probability,  while  the  other  rejection  numbers  considered  as  compatible  with  the  distribution  make  up 
the  remaining  95.9%.  A  more  conservative  decision  would  consider  even  7  rejections  as  falling  into  the  high 
probability  regioa  This  new  high  probability  region  now  consists  of  about  98%  of  the  total  probability. 

Next,  consider  Fig.  8  where  the  future  sample  size  has  been  increased  to  N  =  50.  This  probability  distribution 
is  more  symmetrical  than  the  previous  one  which  was  based  on  a  smaller  sample  size.  The  most  probable  rejection 
frequency  count  is  17,  again  as  intuition  would  suggest,  with  a  probability  of  11.21%.  This  is  the  value  of  z  that 
was  calculated  in  the  previous  section.  Nearby  values  are  also  quite  probable,  dropping  off  somewhat  like  the 
normal  curve,  although  it  must  be  remembered  that  this  is  a  discrete  probability  distribution.  Just  a  casual  visual 
inspection  of  the  bar  graph  of  probabilities  tells  us  that  rejection  frequencies  lower  than  about  8  and  greater  than 
about  28  are  improbable. 

Just  as  before,  we  would  like  to  identify  two  values  of  z,  one  value  at  the  lower  end  and  one  value  at  the  upper 
end,  that  together  demarcate  about  95%  of  the  total  probabUity.  For  example,  in  Fig.  8  for  the  N  =  50  case,  if 
z  =  10  at  the  lower  end  and  z  =  25  at  the  upper  end,  the  total  probabUity  for  all  z  values  in  between  these  two 
values  is  just  about  95%.  On  the  basis  of  this  analysis,  we  would  say  that  obtaining  1 1, 12,  •  •  • ,  or  24  rejections 
from  a  future  sample  of  50  candidates  tested  would  be  compatible  with  the  distribution,  but  that  0  to  10  rejections 
or  25  to  50  rejections  would  be  an  extreme  number. 
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Figure  8:  The  predictive  probability  graph  showing  the  probability  of  the  number  of  rejections  for  the  selection 
system  for  a  sample  size  of  N  =  50.  This  probability  distribution  is  based  on  past  empirical  data  of  172  rejections 
out  of  495  candidates  tested  and  a  theoretical  judgment  of  the  rejection  rate  based  on  a  beta  distribution  with  a  mean 
of  .38. 

Figure  9  can  be  interpreted  in  a  similar  manner.  Figure  9  shows  the  bar  graph  of  the  predictive  probability  for 
the  largest  sample  size  treated,  N  —  100.  The  graph  points  to  frequency  counts  in  the  range  of  31  to  39  as  the 
most  probable.  Together  these  rejection  numbers  contribute  over  61%  to  the  total  distribution.  As  the  sample  size 
N  becomes  larger,  the  influence  of  the  posterior  distribution  shown  in  Fig.  6  becomes  more  obvious.  Even  a  quick 
glance  at  this  distribution  tells  us  that  fiequency  counts  below  20  and  above  50  are  rare.  For  the  more  refined 
analysis,  we  can  pick  out  z  =  24  and  z  —  46  as  the  demarcation  fiequency  counts  that  together  chop  off  about  5% 
of  the  distribution.  We  would  be  surprised,  for  example,  to  see  20  rejections  or  50  rejections  from  the  next  sample 
of  100  candidates,  but  not  surprised  to  observe  25  rejections  or  44  rejections  from  the  same  sample  size. 

Using  the  predictive  distribution  as  a  signal  of  an  external  change 

If  fJ-Pass  and  UFaii  were  completely  stable,  then  we  would  expect  the  posterior  distribution  to  converge  upon  a 
single  rejection  rate  and  a  single  attrition  rate  over  time.  As  the  actual  frequency  data  for  the  number  of  rejections 
and  the  number  of  attritions  accumulates,  the  posterior  probability  distribution  becomes  more  and  more  peaked 
around  one  value  of  0. 

For  example,  imagine  that  n  =  10, 000  candidates  had  been  tested  over  long  period  of  time  and  that  y  =  3800 
of  these  candidates  had  been  rejected  as  predicted  failures  by  the  test  battery.  The  posterior  distribution  would  look 
like  the  curve  depicted  in  Fig.  10.  Here  we  see  a  very  narrow  spike  centered  on  E(0r\y,  n ,  a,/3)  =  .38.  Then  we 
would  be  more  confident  that  Fig.  1  portrays  the  true  state  of  affairs  with  fiFaii  located  at  —0.10  and  y-paas 
located  at  +0.90. 

Likewise,  using  the  same  numerical  example  as  above,  after  the  3800  candidates  had  been  rejected  by  the  test 
battery,  6200  candidates  would  have  been  predicted  passes  and  therefore  entered  training  Let  us  assume  that  of 
these  6200  who  entered  training,  1240  attrited  for  an  estimated  attrition  rate  of  20%.  Technically,  we  would  have  a 
posterior  probability  distribution  for  6a  very  closely  centered  around  .20.  These  frequency  counts  for  the  number  of 
attritions  add  to  the  weight  of  evidence  provided  by  the  rejection  counts  concerning  the  location  of  fj,pa3S  and 
UFaii  as  shown  in  Fig.  1. 
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Figure  9:  The  predictive  probability  graph  showing  the  probability  of  the  number  of  rejections  for  the  selection 
system  for  a  sample  size  of  IV  =  100.  This  probability  distribution  is  based  on  past  empirical  data  of  172  rejections 
out  of  495  candidates  tested  and  a  theoretical  judgment  of  the  rejection  rate  based  on  a  beta  distribution  with  a  mean 

of  .38. 


As  the  amount  of  data  increases,  the  posterior  distributions  for  0T  and  0a  become  even  more  narrowly  peaked 
around  a  given  value.  In  the  example  above,  8r  becomes  more  peaked  around  the  value  of  .38.  In  the  limit  of  an 
infinite  amount  of  data,  the  posterior  distribution  becomes  a  delta  function. 


p{8r\y, n, a,  p)  =  6{6r  -  -38) 

(8) 

and  the  predictive  distribution  collapses  to. 

P(z|lV,  y ,  n,  a,  (3)  =  f  d6r  L(z\0r,  N)  6{6r  -  .38) 

Jo 

(9) 

=  L(z\6r  =  -38,  JVj 

(10) 

For  example,  if  one  is  interested  in  a  future  sample  size  of  100  and  wants  to  know  the  probability  of  obtaining  30 
rejections,  then 

L(z  =  3O|0r  =  .38,  N  =  100)  = 


We  have  completely  eliminated  the  uncertainty  about  the  location  of  fipass  and  PFaii  due  to  the  small  sample  size 
that  we  were  forced  to  contend  with  during  R&D.  The  only  concern  with  sample  size  now  is  the  size  of  the  future 
sample  N  and  that  uncertainty  is  accurately  reflected  in  the  predictive  distribution.  This  rests  on  the  supposition 
that  HPa,s  and  UFaii  remain  static.  On  the  contrary,  events  in  the  real  world  might  cause  dynamic  changes  to 
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Figure  10:  How  the  posterior  distribution  can  become  a  very  narrow  spike  after  a  large  amount  of  empirical  data 
This  posterior  distribution  represents  very  little  uncertainty  about  the  value  of  .38  for  the  rejection  rate  parameter. 


(J'Pass  and  fj-Faii ■  Any  disruption  to  (ipaSs  and  fiFaii  would  translate  into  changes  for  the  number  of  rejections  and 
attritions. 

Similarly,  npaSs  and  HFa.u  could  be  impacted  by  real-world  events  that  cause  them  to  move  further  apart  or 
move  closer  together.  They  do  not  necessanly  have  to  move  in  concert.  For  example,  the  mean  of  the  PASS  group 
could  be  moving  down  and  the  mean  of  the  FAIL  group  could  be  moving  up,  and  vice  versa.  These  changpg  are 
not  due  to  the  small  sample  sizes  discussed  at  the  beginning  of  the  report.  The  uncertainty  attending  small  sample 
size  was  integrated  into  the  prior  probability  distribution  The  present  uncertainty  is  engendered  by  external 
perturbations  such  as  hardware  malfunctions,  lack  of  calibration,  new  computer  operating  systems,  change  in  skill 
level  of  the  candidate  pool,  or  changes  in  the  training  criterion,  to  mention  just  a  few  possible  causes. 

We  would  like  to  have  some  signal  that  would  alert  us  to  these  kinds  of  major  perturbations  to  npass  and 
tiFau-  We  suggest  that  a  low  probability  frequency  count  from  either  of  the  predictive  probability  distributions 
serve  as  that  warning  signal.  For  example,  in  Fig.  8  we  constructed  the  predictive  probability  distribution  for  0r  for 
AT  =  50  based  on  the  prior  distribution  and  some  empirical  data.  Sty  that  after  a  half  year  of  testing  we  found  that 
the  selection  system  had  rejected  26  candidates.  Twenty  six  candidates  is  an  extremely  high  number  of  rejections 
and  happens  to  fell  into  the  upper  2  1/2%  region  of  the  predictive  probability  distribution.  Of  the  24  candidates 
who  did  enter  training,  4  were  attrited. 

A  plausible  scenario  explaining  this  large  number  of  rejections  is  that  fiPass  and  fiFau  have  been  affected  by 
some  external  cause.  If  fxpass  is  shifted  downwards  to  +0.50  and  UFaii  is  shifted  downwards  by  the  samp  amount 
to  -0.50,  then  the  estimated  rejection  numbers  and  attrition  numbers  from  this  model  are  compatible  with  the 
actual  observed  data.  Look  at  Fig.  11  which  shows  this  downward  movement  of  fiPass  and  fipaii-  According  to 
this  model,  26  rejections  and  4  attritions  are  now  to  be  expected.  It  could  be  that  the  external  perturbation  was  an 
overall  degradation  in  the  candidates’s  skill  level.  Another  possibility  is  that  insertion  of  a  new  A/D  card 

inadvertently  made  some  tasks  harder,  causing  every  candidate  to  score  lower  although  their  skill  level  remained 
unchanged. 

The  main  point  is  that  an  anomalous  number  of  rejections  or  attritions,  as  judged  by  the  predictive  probability 
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Figure  11:  A  plausible  downward  movement  in  the  mean  composite  scores  for  the  PASS  and  FAIL  groups  to 
explain  the  abnormally  high  number  of  rejections.  This  shift  in  the  means  represents  the  impact  of  some  external 
perturbation. 
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distribution,  should  be  investigated  and,  if  possible,  the  cause  of  the  external  pertuibation  should  be  tracked  dowa 
This  is  the  quality  control  feature  of  the  predictive  distribution  that  alerts  us  to  the  fact  that  the  locations  of  (ipass 
and  UFaii  may  be  shifting.  On  the  other  hand,  if  the  number  of  rejections  and  attritions  remains  compatible  with 
the  predictive  distribution,  then  the  uncertainly  about  the  non  shifting  locations  of  npass  and  UFaii  is  being 
reduced. 
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